Using skrape{it} to skrap from a JS driven websites - android-studio

I have this kotlin code in android studio that I got from the official documentation of skrape{it}
fun main() {
val scrapedData = skrape {
url = "http://some.url"
mode = Mode.DOM // <--- here's the magic
extract {
div {
withClass = "dynamic"
findFirst { text }
}
}
}
println(scrapedData)
}
and it doesn't even run or build.
I added some import
import ch.qos.logback.classic.spi.CallerData.extract
import it.skrape.fetcher.skrape
import org.jsoup.select.Collector.findFirst
I have this error message
None of the following functions can be called with the arguments supplied:
public fun <R, T> skrape(fetcher: BlockingFetcher<TypeVariable(R)>, init: suspend Scraper<TypeVariable(R)>.() -> TypeVariable(T)): TypeVariable(T) defined in it.skrape.fetcher
public suspend fun <R, T> skrape(fetcher: NonBlockingFetcher<TypeVariable(R)>, init: suspend Scraper<TypeVariable(R)>.() -> TypeVariable(T)): TypeVariable(T) defined in it.skrape.fetcher

Related

What changes are required to make this class scan Barcode in any direction. It works only for QR codes

I am developing a Qr Code and Barcode Scanner App, using CameraX and Zxing but the following class only works for Qr Code. I want to scan barcodes as well from any orientation on an android device
Code Scanning class:
public class QRCodeImageAnalyzer implements ImageAnalysis.Analyzer {
private QRCodeFoundListener listener;
public QRCodeImageAnalyzer(QRCodeFoundListener listener) {
this.listener = listener;
}
#SuppressLint("UnsafeOptInUsageError")
#Override
public void analyze(#NonNull ImageProxy imageProxy) {
Image image = imageProxy.getImage();
if (image.getFormat() == YUV_420_888 || image.getFormat() == YUV_422_888 || image.getFormat() == YUV_444_888) {
ByteBuffer byteBuffer = image.getPlanes()[0].getBuffer();
byte[] imageData = new byte[byteBuffer.capacity()];
byteBuffer.get(imageData);
PlanarYUVLuminanceSource source = new PlanarYUVLuminanceSource(
imageData,
image.getWidth(), image.getHeight(),
0, 0,
image.getWidth(), image.getHeight(),
false
);
BinaryBitmap binaryBitmap = new BinaryBitmap(new HybridBinarizer(source));
try {
Result result = new QRCodeMultiReader().decode(binaryBitmap);
listener.onQRCodeFound(result.getText());
} catch (FormatException | ChecksumException | NotFoundException e) {
listener.qrCodeNotFound();
}
}
imageProxy.close();
}
}
I also tried this class CameraX with MLKit: I tried the sample from the official docs of MLKit provided in the first answer but it scan nothing neither QR Code nor Barcode. Please take a look if I put things the wrong way.
import android.annotation.SuppressLint
import android.util.Log
import androidx.camera.core.ExperimentalGetImage
import androidx.camera.core.ImageAnalysis
import androidx.camera.core.ImageProxy
import com.google.mlkit.vision.barcode.Barcode
import com.google.mlkit.vision.barcode.BarcodeScannerOptions
import com.google.mlkit.vision.barcode.BarcodeScanning
import com.google.mlkit.vision.common.InputImage
#UseExperimental(markerClass = [ExperimentalGetImage::class])
class QrCodeAnalyzer(
private val onQrCodesDetected: (qrCodes: List<Barcode>?) -> Unit
) : ImageAnalysis.Analyzer {
private val TAG: String = "Kamran"
private fun rotationDegreesToFirebaseRotation(rotationDegrees: Int): Int {
return when (rotationDegrees) {
0 -> 0
90 -> 1
180 -> 2
270 -> 3
else -> throw IllegalArgumentException("Not supported")
}
}
#SuppressLint("UnsafeOptInUsageError")
override fun analyze(image: ImageProxy) {
val rotation = rotationDegreesToFirebaseRotation(image.imageInfo.rotationDegrees)
image.image?.let{
val optionss = BarcodeScannerOptions.Builder()
.setBarcodeFormats(
Barcode.FORMAT_QR_CODE,
Barcode.FORMAT_EAN_8,
Barcode.FORMAT_EAN_13)
.build()
val imageValue = InputImage.fromMediaImage(it, image.imageInfo.rotationDegrees)
//val options = BarcodeScannerOptions.Builder().setBarcodeFormats(Barcode.FORMAT_QR_CODE).build()
val scanner = BarcodeScanning.getClient(optionss)
scanner.process(imageValue)
.addOnCompleteListener { barcodes ->
barcodes.result?.forEach { barcode ->
val bounds = barcode.boundingBox
val corners = barcode.cornerPoints
val rawValue = barcode.rawValue
}
onQrCodesDetected(barcodes.result)
image.image?.close()
image.close()
Log.d(TAG, "Successfully got inside onCompleteListener")
}
.addOnFailureListener { failure ->
failure.printStackTrace()
image.image?.close()
image.close()
}
}
}
}
I am not sure If using ZXing android library is a requirement (afaik it is no longer developed). If it is not a hard requirement, I would suggest you to give it a try for MLKit Barcode Scanner. It scans QR as well.
Also, you would probably find it easier to implement with CameraX as they already have multiple examples and sample code.
For example check this guide for step by step instructions.
Particularly this part.
You can configire the scanner first:
val options = BarcodeScannerOptions.Builder()
.setBarcodeFormats(
Barcode.FORMAT_QR_CODE,
Barcode.FORMAT_EAN_8,
Barcode.FORMAT_EAN_13)
.build()
then;
private class YourImageAnalyzer : ImageAnalysis.Analyzer {
override fun analyze(imageProxy: ImageProxy) {
val mediaImage = imageProxy.image
if (mediaImage != null) {
val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees)
// Pass image to an ML Kit Vision API
// ...
}
}
}
finally this is how you get it decoded:
val result = scanner.process(image)
.addOnSuccessListener { barcodes ->
// Task completed successfully
// ...
}
.addOnFailureListener {
// Task failed with an exception
// ...
}

Get return value from thread, is this Kotlin code thread safe?

I would like to run some treads, wait till all of them are finished and get the results.
Possible way to do that would be in the code below. Is it thread safe though?
import kotlin.concurrent.thread
sealed class Errorneous<R>
data class Success<R>(val result: R) : Errorneous<R>()
data class Fail<R>(val error: Exception) : Errorneous<R>()
fun <R> thread_with_result(fn: () -> R): (() -> Errorneous<R>) {
var r: Errorneous<R>? = null
val t = thread {
r = try { Success(fn()) } catch (e: Exception) { Fail(e) }
}
return {
t.join()
r!!
}
}
fun main() {
val tasks = listOf({ 2 * 2 }, { 3 * 3 })
val results = tasks
.map{ thread_with_result(it) }
.map{ it() }
println(results)
}
P.S.
Are there better built-in tools in Kotlin to do that? Like process 10000 tasks with pool of 10 threads?
It should be threads, not coroutines, as it will be used with legacy code and I don't know if it works well with coroutines.
Seems like Java has Executors that doing exactly that
fun <R> execute_in_parallel(tasks: List<() -> R>, threads: Int): List<Errorneous<R>> {
val executor = Executors.newFixedThreadPool(threads)
val fresults = executor.invokeAll(tasks.map { task ->
Callable<Errorneous<R>> {
try { Success(task()) } catch (e: Exception) { Fail(e) }
}
})
return fresults.map { future -> future.get() }
}

Kotlin how to combine different receivers via with clause without repeating it

I have Parent-Search-Child system as below:
class Room
class Building {
fun find(by: By) = Room()
}
sealed class By {
abstract fun search(): Room
class ById(id: String) : By() {
override fun search(): Room = Room() // epic search method
}
class ByName(name: String) : By() {
override fun search(): Room = Room() // epic search method
}
class Byurpose(purpose: String) : By() {
override fun search(): Room = Room() // epic search method
}
companion object {
fun id(id: String) = ById(id)
fun name(name: String) = ByName(name)
fun purpose(purpose: String) = Byurpose(purpose)
}
}
Which can be used as follows:
val building = Building()
val room = building.find(By.name("Toilet"))
However, I am not very satisfied with the current syntax, which could be much less verbose in Kotlin. In addition, building.find can appear thousands times in the code. I could implement it differently, but actually I don't own Room, Building or By classes, so I can't. Thus this was my approach:
I implemented context class that stores Building reference, and use it internally as source for search methods:
class BuildingContext(private val building: Building) {
fun String.findById() = building.find(By.id(this))
fun String.findByName() = building.find(By.name(this))
fun String.findByPurpose() = building.find(By.purpose(this))
}
It can be used as below:
with(BuildingContext(building)) {
val room2 = "Toilet".findByName()
}
After that I noticed that I only use one search method in 99% cases, so (for sake of even shorter syntax!) I implemented following classes:
object AlwaysSearchById {
fun String.find(building: Building) = building.find(By.id(this))
}
object AlwaysSearchByName {
fun String.find(building: Building) = building.find(By.name(this))
}
object AlwaysSearchByPurpose {
fun String.find(building: Building) = building.find(By.purpose(this))
}
Which can be used this way:
with(AlwaysSearchByName) {
val room3 = "Toilet".find(building)
}
Unfortunately, building reference appears again. The ideal syntax would be "Toilet".find(). I could fix it re-implementing Always~ classes as follows:
class AlwaysSearchByNameV2(private val building: Building) {
fun String.find() = building.find(By.name(this))
}
And it would be used as below:
with(AlwaysSearchByNameV2(building)) {
val room = "Toilet".find()
}
But it some cases, I would like to access BuildingContext methods as well, so So I have to write:
with(BuildingContext(building)) {
with(AlwaysSearchByNameV2(building)) {
val toilet = "Toilet".find()
val randomRoom = "123".findById()
}
}
The question is - How I reduce multiple with clauses in this case?
In the example above there are only 2 with clauses, but it's only basic example. In real world there could be dozens of them, and writing with(with(with(with(with... would surely be a pain.
On the side note this doesn't work:
with(BuildingContext(building), AlwaysSearchByNameV2(building)) {
val toilet = "Toilet".find()
val randomRoom = "123".findById()
}
nor this
with(*arrayOf(BuildingContext(building), BuildingContext(building))) {
val toilet = "Toilet".find()
val randomRoom = "123".findById()
}
You can write custom scoping functions instead of relying on with all the time. For example you can add an extension function that will run a block of code in scope of AlwaysSearchByNameV2 object:
inline fun BuildingContext.byName(f : AlwaysSearchByNameV2.() -> Unit) = AlwaysSearchByNameV2(building).apply(f)
And use it:
with(BuildingContext(building)) { // this: BuildingContext
byName { // this: AlwaysSearchByNameV2
val toilet = "Toilet".find()
val randomRoom = "123".findById() // can still refer to BuildingContext
}
// back to this: BuildingContext
}

How passing a protocol as parameter in Swift

In Objective-C, I know how passing a protocol as parameter:
- (void)MyMethod:(Protocol *)myparameter
But in Swift there is no more Protocol type.
How can I pass a protocol as parameter without knowing which is ?
In one of your comments you say:
"I want create a method which return an array of type of class which implements a desired protocol."
Have you tried something like the following:
//notice the use of #objc here
#objc protocol AlertProtocol
{
func getMyName()->String
}
class Class1 : AlertProtocol
{
let name = "Object 1"
func getMyName() -> String
{
return name
}
}
class Class2 : AlertProtocol
{
let name = "Object 2"
func getMyName() -> String
{
return name
}
}
//borrowing from and refactoring siLo's answer
func classesConformingToProtocol(proto:Protocol) -> [AnyClass]
{
let availableClasses : [AnyClass] = [ Class1.self, Class2.self ]
var conformingClasses = Array<AnyClass>()
for myClass : AnyClass in availableClasses
{
if myClass.conforms(to: proto)
{
conformingClasses.append(myClass)
}
}
return conformingClasses
}
Then use the above structure like this:
let classes = classesConformingToProtocol(AlertProtocol.self)
The tricky part that does the work is the "#objc" that exposes the protocol to the objective c runtime and allows us to pass any "Protocol Type" as a parameter.
Probably at some point in the future we will be able to do this in a "pure" Swift way.
Here is what I have tried:
#objc protocol Walker
{
func walk()
}
#objc protocol Runner
{
func run()
}
#objc class Zombie : Walker
{
func walk () { println("Brains...") }
}
#objc class Survivor : Runner
{
func run() { println("Aaaah, zombies!") }
}
func classesConformingToProtocol(proto:Protocol) -> AnyClass[]
{
let availableClasses : AnyClass[] = [ Zombie.self, Survivor.self ]
var conformingClasses = Array<AnyClass>()
for myClass : AnyClass in availableClasses
{
if myClass.conformsToProtocol(proto)
{
conformingClasses.append(myClass)
}
}
return conformingClasses
}
// This does not work
let walkers = classesConformingToProtocol(Walker.self)
let runners = classesConformingToProtocol(Runner.self)
I have been unable to convert Swift's Metatype information into a Protocol object.
In swift 2.0, I use it like this before:
classA.conformsToProtocol(XXXProtocol.self as! Protocol)
It doesn't works fine...
Look the definition of Protocol:
// All methods of class Protocol are unavailable.
// Use the functions in objc/runtime.h instead.
#available(iOS 2.0, *)
public class Protocol {
}
All are unavailable...and I don't know which to use instead in objc/runtime.h
So I have to use this method:
if ClassA is protocol<XXXProtocol> {
// do something
}
Currently, it works...
If you don't allow use #objc (because yours protocols have property, for example), the only solution that I found is with closure. Then, you need use a closure to use a protocol and return a value.
protocol Proto { }
protocol Proto2 { }
class Foo: Proto { }
class Bar: Proto, Proto2 { }
class Baz: Proto2 { }
class Qux { }
func printConforms(classList: [AnyClass], protoCond: (AnyClass) -> Any?) {
for i in classList {
print(i, terminator: " -> ")
if protoCond(i) != nil {
print("is subscriber")
} else {
print("NOT IS subscriber")
}
}
}
let myClasses: [AnyClass] = [Foo.self, Bar.self, Baz.self, Qux.self]
printConforms(classList: myClasses, protoCond: { $0 as? Proto.Type })
More complete example: https://gist.github.com/brunomacabeusbr/eea343bb9119b96eed3393e41dcda0c9
Edit
Another better solution is using generics, for example:
protocol Proto { }
class Foo: Proto { }
class Bar: Proto { }
class Baz { }
func filter<T>(classes: [AnyClass], byConformanceTo: T.Type) -> [AnyClass] {
return classes.filter { $0 is T }
}
filter(classes: [Foo.self, Bar.self, Baz.self], byConformanceTo: Proto.Type.self)
// return [Foo.self, Bar.self]
Worked out a way today (Xcode 6.1):
Firstly, the protocol must be marked as #objc for any checking to work.
Then use an "if let" cast to check for conformance.
#objc protocol MyProtocol {
var protocolValue: Int { get set }
}
if let conformingObject = someObject as? MyProtocol {
// conformingObject is now someObject cast to MyProtocol
conformingObject.protocolValue = 3
}

Coercion befuddlement in Groovy

Why does the following
class Test {
#Test
void go() {
def foo1 = new MockFoo1() as Foo
def foo2 = new MockFoo2() as Foo
}
interface Foo {}
class MockFoo1 {}
class MockFoo2 {}
}
Result in a java.lang.IllegalArgumentException: argument type mismatch on the foo2 coercion?
This only happens if I coerce 2 objects of 2 different types to the same interface during a single path of execution. The groovy approved way of using closures or maps to achieve this kind of duck typing works fine.
Any light shed appreciated.
It's a bug with the ProxyGenerator adapterCache. As a workaround, you can also use some Groovy trickery to make this work:
interface Foo {
static a = {
[MockFoo1, MockFoo2].each {
it.metaClass.asType = { Class klazz ->
try {
DefaultGroovyMethods.asType(delegate, klazz)
} catch (e) {
def cache = ProxyGenerator.INSTANCE.#adapterCache.#cache
cache.each { k, v ->
cache.remove(k)
}
DefaultGroovyMethods.asType(delegate, klazz)
}
}
}
}()
}
class MockFoo1 {}
class MockFoo2 {}
def a = new MockFoo1() as Foo
def b = new MockFoo2() as Foo
assert a instanceof Foo
assert b instanceof Foo
Hope this helps!

Resources