Related
I am new to apache spark and am trying to run a custom nearest neighbor algorithm on an RDD that has been partitioned into 2 parts using a custom partitioner. The JavaPairRDD contains the graph details and the random object created on the graph.
According to my logic, I am building subgraphs for each partition, and I am running a custom algorithm on each subgraph. It seems to be working "although not properly". I am not sure if this is the correct way to apply action in each partition. I am adding my code and the results as well. Comments and suggestions are highly appreciated.
// <Partition_Index_Key, Map<Source_vertex, Map<Destination Vertex, Tuple2<Edge_Length, ArrayList of Random Objects>>
JavaPairRDD<Object, Map<Object, Map<Object, Tuple2<Double, ArrayList<RoadObject>>>>> adjVertForSubgraphsRDD = jscontext
.parallelizePairs(adjacentVerticesForSubgraphs)
.partitionBy(new CustomPartitioner(CustomPartitionSize));
//applying foreachPartition action on JavaPairRDD
adjVertForSubgraphsRDD.foreachPartition(
new VoidFunction<Iterator<Tuple2<Object, Map<Object, Map<Object, Tuple2<Double, ArrayList<RoadObject>>>>>>>() {
/**
*
*/
private static final long serialVersionUID = 1L;
#Override
public void call(
Iterator<Tuple2<Object, Map<Object, Map<Object, Tuple2<Double, ArrayList<RoadObject>>>>>> tupleRow)
throws Exception {
int sourceVertex;
int destVertex;
double edgeLength;
int roadObjectId;
boolean roadObjectType;
double distanceFromStart;
CoreGraph subgraph0 = new CoreGraph();
CoreGraph subgraph1 = new CoreGraph();
while (tupleRow.hasNext()) {
Map<Object, Map<Object, Tuple2<Double, ArrayList<RoadObject>>>> newMap = tupleRow.next()
._2();
if ((Integer.parseInt(String.valueOf(tupleRow.next()._1())) == 0)) {
for (Object srcVertex : newMap.keySet()) {
for (Object dstVertex : newMap.get(srcVertex).keySet()) {
if (newMap.get(srcVertex).get(dstVertex)._2() != null) {
sourceVertex = Integer.parseInt(String.valueOf(srcVertex));
destVertex = Integer.parseInt(String.valueOf(dstVertex));
edgeLength = newMap.get(srcVertex).get(dstVertex)._1();
subgraph0.addEdge(sourceVertex, destVertex, edgeLength);
for (int i = 0; i < newMap.get(srcVertex).get(dstVertex)._2()
.size(); i++) {
int currentEdgeId = subgraph0.getEdgeId(sourceVertex, destVertex);
roadObjectId = newMap.get(srcVertex).get(dstVertex)._2().get(i)
.getObjectId();
roadObjectType = newMap.get(srcVertex).get(dstVertex)._2().get(i)
.getType();
distanceFromStart = newMap.get(srcVertex).get(dstVertex)._2().get(i)
.getDistanceFromStartNode();
RoadObject rn0 = new RoadObject();
rn0.setObjId(roadObjectId);
rn0.setType(roadObjectType);
rn0.setDistanceFromStartNode(distanceFromStart);
subgraph0.addObjectOnEdge(currentEdgeId, rn0);
}
} else {
sourceVertex = Integer.parseInt(String.valueOf(srcVertex));
destVertex = Integer.parseInt(String.valueOf(dstVertex));
edgeLength = newMap.get(srcVertex).get(dstVertex)._1();
subgraph0.addEdge(sourceVertex, destVertex, edgeLength);
}
}
}
} else if ((Integer.parseInt(String.valueOf(tupleRow.next()._1())) == 1)) {
for (Object srcVertex : newMap.keySet()) {
for (Object dstVertex : newMap.get(srcVertex).keySet()) {
if (newMap.get(srcVertex).get(dstVertex)._2() != null) {
sourceVertex = Integer.parseInt(String.valueOf(srcVertex));
destVertex = Integer.parseInt(String.valueOf(dstVertex));
edgeLength = newMap.get(srcVertex).get(dstVertex)._1();
subgraph1.addEdge(sourceVertex, destVertex, edgeLength);
for (int i = 0; i < newMap.get(srcVertex).get(dstVertex)._2()
.size(); i++) {
int currentEdgeId = subgraph1.getEdgeId(sourceVertex, destVertex);
roadObjectId = newMap.get(srcVertex).get(dstVertex)._2().get(i)
.getObjectId();
roadObjectType = newMap.get(srcVertex).get(dstVertex)._2().get(i)
.getType();
distanceFromStart = newMap.get(srcVertex).get(dstVertex)._2().get(i)
.getDistanceFromStartNode();
RoadObject rn1 = new RoadObject();
rn1.setObjId(roadObjectId);
rn1.setType(roadObjectType);
rn1.setDistanceFromStartNode(distanceFromStart);
subgraph1.addObjectOnEdge(currentEdgeId, rn1);
}
} else {
sourceVertex = Integer.parseInt(String.valueOf(srcVertex));
destVertex = Integer.parseInt(String.valueOf(dstVertex));
edgeLength = newMap.get(srcVertex).get(dstVertex)._1();
subgraph1.addEdge(sourceVertex, destVertex, edgeLength);
}
}
}
}
}
// Straight forward nearest neighbor algorithm from each true to false.
ANNNaive ann = new ANNNaive();
System.err.println("-------------------------------");
Map<Integer, Integer> nearestNeighorPairsSubg0 = ann.compute(subgraph0, true);
System.out.println("for subgraph0");
System.out.println(nearestNeighorPairsSubg0);
System.err.println("-------------------------------");
System.err.println("-------------------------------");
Map<Integer, Integer> nearestNeighorPairsSubg1 = ann.compute(subgraph1, true);
System.out.println("for subgraph1");
System.out.println(nearestNeighorPairsSubg1);
System.err.println("-------------------------------");
}
});
Struggling with some Revit code to copy the profile of a floor and use it as the sketch profile for a void extrusion in a family.
Here is the Full Sharp Development Code. It half works in my custom project template, when I try to use it in an out of the box project generated from revit default template it gives the error "a managed exception was thrown by revit or by one of its external applications"
In my template it cannot properly split the curves into a secondary array. It says the array elements are being changed but when the loop runs again the element i is back to it's original content??? The TaskDialog clearly says the elements have changed, until the loop iterates again.
Full code: To work it requires a generic family with the name "Void - Custom" to be in the project. The "If found" near the bottom last page and a half of code, is where the for loop is not behaving as expected.
/*
* Created by SharpDevelop.
* User: arautio
* Date: 4/30/2019
* Time: 11:10 AM
*
* To change this template use Tools | Options | Coding | Edit Standard Headers.
*/
using System;
using Autodesk.Revit.UI;
using Autodesk.Revit.DB;
using Autodesk.Revit.DB.Architecture;
using Autodesk.Revit.DB.Structure;
using Autodesk.Revit.UI.Selection;
using System.Collections.Generic;
using System.Linq;
using Autodesk.Revit.ApplicationServices;
using Autodesk.Revit.Attributes;
using System.Text;
using System.IO;
using System.Diagnostics;
namespace ARC
{
[Autodesk.Revit.Attributes.Transaction(Autodesk.Revit.Attributes.TransactionMode.Manual)]
[Autodesk.Revit.DB.Macros.AddInId("3411F411-6FC1-4A4D-9CFD-37ABB2028A15")]
public partial class ThisApplication
{
private void Module_Startup(object sender, EventArgs e)
{
}
private void Module_Shutdown(object sender, EventArgs e)
{
}
#region Revit Macros generated code
private void InternalStartup()
{
this.Startup += new System.EventHandler(Module_Startup);
this.Shutdown += new System.EventHandler(Module_Shutdown);
}
#endregion
public void FloorGrating()
{
StringBuilder sb = new StringBuilder();
Dictionary<Floor, List<ModelCurve>> dict_SketchLines = new Dictionary<Floor, List<ModelCurve>>();
UIDocument uidoc = this.ActiveUIDocument;
Document document = uidoc.Document;
View activev = document.ActiveView;
ElementId levelId = null;
levelId = activev.LevelId;
Element levelem = document.GetElement( levelId );
Level lev = document.ActiveView.GenLevel;
Reference refsel = uidoc.Selection.PickObject(ObjectType.Element, "Select Floor to Add Grating To");
Element elem = document.GetElement(refsel.ElementId);
Floor f = elem as Floor;
List<ElementId> _deleted = null;
using (Transaction t = new Transaction(document, "temp"))
{
t.Start();
document.Regenerate();
_deleted = document.Delete(elem.Id).ToList();
t.RollBack();
}
bool SketchLinesFound = false;
List<ModelCurve> _sketchCurves = new List<ModelCurve>();
foreach (var id in _deleted)
{
ModelCurve mc = document.GetElement(id) as ModelCurve;
if (mc != null)
{
_sketchCurves.Add(mc);
SketchLinesFound = true;
}
else
{
if (SketchLinesFound) break;
}
}
dict_SketchLines.Add(f, _sketchCurves);
foreach (Floor key in dict_SketchLines.Keys)
{
List<ModelCurve> _curves = dict_SketchLines[key];
sb.AppendLine(string.Format("floor {0} has sketchlines:", key.Id));
foreach (ModelCurve mc in _curves)
{
sb.AppendLine(string.Format("{0} <{1}>", mc.GetType(), mc.Id));
sb.AppendLine(string.Format("<{0}>", mc.GeometryCurve.IsBound.ToString()));
if (mc.GetType().ToString() == "Autodesk.Revit.DB.ModelArc" && mc.GeometryCurve.IsBound == false)
{
TaskDialog.Show("Revit", "Circle Found");
}
try
{
sb.AppendLine(string.Format("<{0} -- {1}>", mc.GeometryCurve.GetEndPoint(0), mc.GeometryCurve.GetEndPoint(1)));
}
catch
{
}
}
sb.AppendLine();
}
//TaskDialog.Show("debug", sb.ToString());
Document docfamily;
Family fam;
string ftitle = document.Title;
string fpath = document.PathName;
int ftitlelen = ftitle.Length + 4;
int fpathlen = fpath.Length;
int finpathlen = fpathlen - ftitlelen;
string sfinpath = fpath.Substring(0,finpathlen);
string famname = "GratingVoid";
string fext = ".rfa";
int counter = 1;
while (counter < 100)
{
famname = ("GratingVoid" + counter as String);
Family family = FindElementByName(document,typeof(Family),famname)as Family;
if( null == family )
{
sfinpath = (sfinpath + famname + fext);
counter = 1000;
}
counter += 1;
}
FilteredElementCollector collector0 = new FilteredElementCollector(document);
ICollection<Element> collection0 = collector0.WhereElementIsNotElementType().ToElements();
List<FamilySymbol> fsym0 = new FilteredElementCollector(document).OfClass(typeof(FamilySymbol)).Cast<FamilySymbol>().ToList();
FamilySymbol famsymb0 = null;
foreach (FamilySymbol symb in fsym0)
{
if (symb.Name == "Void - Custom")
{
famsymb0 = symb as FamilySymbol;
}
}
fam = famsymb0.Family;
docfamily = document.EditFamily(fam);
try
{
docfamily.SaveAs(#sfinpath);
}
catch
{
TaskDialog.Show("Revit", "Could Not Save Void Family");
}
using (Transaction trans = new Transaction(docfamily))
{
trans.Start("family");
bool circleflag = false;
ElementId delid = null;
FilteredElementCollector collector = new FilteredElementCollector( docfamily );
foreach(Element element in collector.OfClass(typeof(GenericForm)))
{
delid = element.Id;
}
docfamily.Delete(delid);
CurveArray loccurva = new CurveArray();
foreach (Floor key in dict_SketchLines.Keys)
{
List<ModelCurve> _curves = dict_SketchLines[key];
foreach (ModelCurve mc in _curves)
{
if (mc.GetType().ToString() == "Autodesk.Revit.DB.ModelArc" && mc.GeometryCurve.IsBound == false)
{
circleflag = true;
}
LocationCurve lcurve = mc.Location as LocationCurve;
Curve c = lcurve.Curve as Curve;
loccurva.Append(c);
}
}
try
{
if (circleflag == true && loccurva.Size == 2)
{
Curve tempc;
if (loccurva.get_Item(0).GetType().ToString() == "Autodesk.Revit.DB.Arc")
{
tempc = loccurva.get_Item(0);
}
else
{
tempc = loccurva.get_Item(1);
}
loccurva.Clear();
loccurva.Append(tempc);
}
CurveArrArray newcurarr = new CurveArrArray();
newcurarr.Append(loccurva);
SortCurvesContiguousArray(newcurarr);
TaskDialog.Show("Revit CurveArray Array Size" , newcurarr.Size.ToString());
foreach (CurveArray ca in newcurarr)
{
TaskDialog.Show("Revit CurveArray within Array Size" , ca.Size.ToString());
}
// Below is edited for error control - leaving out the secondary loops for now
CurveArrArray switcharr = new CurveArrArray();
//switcharr.Append(newcurarr.get_Item(1));
switcharr.Append(newcurarr.get_Item(0));
//SortCurvesContiguousArray(loccurva);
//CurveArrArray newcurarr = new CurveArrArray();
//newcurarr.Append(loccurva);
double end = 1;
SketchPlane sketch = FindElementByName( docfamily,typeof( SketchPlane ), "Ref. Level" ) as SketchPlane;
docfamily.FamilyCreate.NewExtrusion(false, switcharr, sketch, end);
}
catch
{
TaskDialog.Show("Revit", "Could Not Write to Curve Array or Create Extrusion");
}
trans.Commit();
}
docfamily.Save();
docfamily.LoadFamily(document, new CustomFamilyLoadOption());
docfamily.Close();
File.Delete(sfinpath);
Family familynew = FindElementByName(document,typeof(Family),famname)as Family;
if( null == familynew )
{
TaskDialog.Show("Revit", "Family Does Not Exist");
}
FilteredElementCollector collector1 = new FilteredElementCollector(document);
ICollection<Element> collection = collector1.WhereElementIsNotElementType().ToElements();
List<FamilySymbol> fsym = new FilteredElementCollector(document).OfClass(typeof(FamilySymbol)).Cast<FamilySymbol>().ToList();
FamilySymbol famsymb = null;
foreach (FamilySymbol symb in fsym)
{
if (symb.Name == famname)
{
famsymb = symb as FamilySymbol;
}
}
using (Transaction trans = new Transaction(document))
{
trans.Start("PlaceVoid");
if( ! famsymb.IsActive )
{
famsymb.Activate();
}
XYZ p = new XYZ(0,0,0);
FamilyInstance gratingvoid = document.Create.NewFamilyInstance( p, famsymb, lev, lev, StructuralType.NonStructural );
document.Regenerate();
trans.Commit();
}
}
//--------------------------------------------------------------------------------------------------------------------------------------------------------
static public Element FindElementByName(Document doc,Type targetType,string targetName)
{
return new FilteredElementCollector( doc ).OfClass( targetType ).FirstOrDefault<Element>(e => e.Name.Equals( targetName ) );
}
//--------------------------------------------------------------------------------------------------------------------------------------------------------
public class CustomFamilyLoadOption : IFamilyLoadOptions
{
public bool OnFamilyFound(bool familyInUse, out bool overwriteParameterValues)
{
overwriteParameterValues = true;
return true;
}
public bool OnSharedFamilyFound(Family sharedFamily,bool familyInUse,out FamilySource source, out bool overwriteParameterValues)
{
source = FamilySource.Family;
overwriteParameterValues = true;
return true;
}
}
//--------------------------------------------------------------------------------------------------------------------------------------------------------
const double _inch = 1.0 / 12.0;
const double _sixteenth = _inch / 16.0;
static Curve CreateReversedCurve(Curve orig )
{
//if( !IsSupported( orig ) )
//{
// throw new NotImplementedException("CreateReversedCurve for type " + orig.GetType().Name );
//}
if( orig is Line )
{
//return creapp.NewLineBound(orig.GetEndPoint( 1 ), orig.GetEndPoint( 0 ) );
return Line.CreateBound(orig.GetEndPoint( 1 ), orig.GetEndPoint( 0 ) );
}
else if( orig is Arc )
{
// return creapp.NewArc( orig.GetEndPoint( 1 ), orig.GetEndPoint( 0 ), orig.Evaluate( 0.5, true ) );
return Arc.Create( orig.GetEndPoint( 1 ), orig.GetEndPoint( 0 ), orig.Evaluate( 0.5, true ) );
}
else
{
throw new Exception(
"CreateReversedCurve - Unreachable" );
}
}
public static void SortCurvesContiguousArray(CurveArrArray curvesarr)
{
double _precision1 = 1.0 / 12.0 / 16.0; // around 0.00520833
double _precision2 = 0.001; // limit for CurveLoop.Create(...)
int cn = curvesarr.Size;
int ci = 0;
while (ci < cn)
{
CurveArray curves = curvesarr.get_Item(ci);
ci +=1;
// account for multiple curve loops with secondary array
CurveArray loop1 = new CurveArray();
CurveArray loop2 = new CurveArray();
int n = curves.Size;
int split = 1;
// Walk through each curve (after the first)
// to match up the curves in order
for (int i = 0; i < n; ++i)
{
TaskDialog.Show("Revit I Loop Run", i.ToString());
Curve curve = curves.get_Item(i);
if (curve.GetType().ToString() == "Autodesk.Revit.DB.Arc" && curve.IsBound == false)
{
break;
}
XYZ beginPoint = curve.GetEndPoint(0);
XYZ endPoint = curve.GetEndPoint(1);
XYZ p,q;
// Find curve with start point = end point
bool found = (i + 1 >= n);
for (int j = i + 1; j < n; ++j)
{
p = curves.get_Item(j).GetEndPoint(0);
q = curves.get_Item(j).GetEndPoint(1);
// If there is a match end->start,
// this is the next curve
if (p.DistanceTo(endPoint) < _precision1)
{
if (p.DistanceTo(endPoint) > _precision2)
{
XYZ intermediate = new XYZ((endPoint.X + p.X) / 2.0, (endPoint.Y + p.Y) / 2.0, (endPoint.Z + p.Z) / 2.0);
curves.set_Item(i, Line.CreateBound(beginPoint, intermediate));
curves.set_Item(j, Line.CreateBound(intermediate, q));
}
if (i + 1 != j)
{
Curve tmp = curves.get_Item(i + 1);
curves.set_Item(i + 1, curves.get_Item(j));
curves.set_Item(j, tmp);
}
found = true;
break;
}
// If there is a match end->end,
// reverse the next curve
if (q.DistanceTo(endPoint) < _precision1)
{
if (q.DistanceTo(endPoint) > _precision2)
{
XYZ intermediate = new XYZ((endPoint.X + q.X) / 2.0, (endPoint.Y + q.Y) / 2.0, (endPoint.Z + q.Z) / 2.0);
curves.set_Item(i, Line.CreateBound(beginPoint, intermediate));
curves.set_Item(j, Line.CreateBound(p, intermediate));
}
if (i + 1 == j)
{
curves.set_Item(i + 1, CreateReversedCurve(curves.get_Item(j)));
}
else
{
Curve tmp = curves.get_Item(i + 1);
curves.set_Item(i + 1, CreateReversedCurve(curves.get_Item(j)));
curves.set_Item(j, tmp);
}
found = true;
break;
}
}
if (!found)
{
// if not found, must be part of a new loop - move it to the back and keep going and add to second array
TaskDialog.Show("Revit No Match Found for item", i.ToString());
TaskDialog.Show("Revit", "Moveing it to back of list");
Curve tmp1 = curves.get_Item(i);
TaskDialog.Show("Revit tmp1 Current i item endpt", tmp1.GetEndPoint(0).ToString());
loop2.Append(tmp1);
Curve tmp2 = curves.get_Item(n - split);
TaskDialog.Show("Revit tmp2 Back of list item endpt", tmp2.GetEndPoint(0).ToString());
// set current item to rear
curves.set_Item(i, tmp2);
// set rear item to current
curves.set_Item(n - split, tmp1);
TaskDialog.Show("Revit new item i endpt", curves.get_Item(i).GetEndPoint(0).ToString());
TaskDialog.Show("Revit moved item endpt", curves.get_Item(n - split).GetEndPoint(0).ToString());
// error testing - try to append in a different manner and check values
//curves.set_Item(i, Line.CreateBound(curves.get_Item(i).GetEndPoint(0), curves.get_Item(i).GetEndPoint(1)));
//curves.set_Item(n - split, Line.CreateBound(curves.get_Item(n - split).GetEndPoint(0), curves.get_Item(n - split).GetEndPoint(1)));
//Curve ncurve = Line.CreateBound(curves.get_Item(n - split).GetEndPoint(0), curves.get_Item(n - split).GetEndPoint(1));
//TaskDialog.Show("Revit Appended to Loop2 Endpoint", ncurve.GetEndPoint(0).ToString());
//loop2.Append(ncurve);
//set the split off counter so items not fitting in first loop can be split to new array.
split += 1;
//reset the counter back so item moved from rear can be checked in next run of for loop
i -= 2;
}
//set counter to end for loop when all items that do not fit in first loop are processed
if (i >= n - (split + 1))
{
TaskDialog.Show("Revit", "End Of Looping");
TaskDialog.Show("Revit - The Split Number", split.ToString());
i = n;
}
}
int counter = 0;
// recreate array with only items from first loop found
while (counter <= (n - split))
{
loop1.Append(curves.get_Item(counter));
counter += 1;
}
TaskDialog.Show("Revit loop1 Size", loop1.Size.ToString());
curvesarr.Clear();
curvesarr.Append(loop1);
if (loop2.Size > 0)
{
string stringinfo = "";
// run the loop detection on a second array that was split from the first
TaskDialog.Show("Revit loop2 Size", loop2.Size.ToString());
CurveArrArray tmpcurvesarr = new CurveArrArray();
tmpcurvesarr.Append(loop2);
SortCurvesContiguousArray(tmpcurvesarr);
loop2.Clear();
loop2 = tmpcurvesarr.get_Item(0);
curvesarr.Append(loop2);
foreach (Curve ccc in loop2)
{
stringinfo = (stringinfo + " " + ccc.GetEndPoint(0).ToString() + " - " + ccc.GetEndPoint(1).ToString());
}
TaskDialog.Show("Revit", stringinfo);
}
}
}
}
}
Thanks for any and all help.
Shane
First you check undeniable signature scheme verifying algorithm.
Than in that there is first step telling to chose two random x1 and x2.
But the verifying algo works sometime and sometime failed depending on two randoms. ( algo implemented in java and correctly).
For the same input if we run that algo multiple time, sometime it signature is matched and sometime not, last comparison is failed.
Plz help me. Im stuck here ( lack of math).
Actually it is an signature scheme and in paper it was confusing but now i've figure out solution.
Solution code is: let me know if you need me to explain this code.
import java.lang.*;
import java.util.*;
import java.math.*;
import java.security.MessageDigest;
import java.nio.file.*;
import java.io.*;
import java.security.*;
class UndeniableSignature {
int Q; // prime
int P; // prime of form P = 2Q + 1
BigInteger G; // Generator of Z*P
int D; // private key belongs to [2 ... Q-1]
BigInteger Y; // G^D (mod P)
int invD; // D^-1 (mod Q)
Random rand = new SecureRandom(); // uniform random generator
public int[] extendedEuclidean(int a, int b) {
if(b==0) {
int[] arr=new int[3];
arr[0] = a;
arr[1] = 1;
arr[2] = 0;
return arr;
}
int[] subResult = extendedEuclidean(b,a%b);
int[] arr=new int[3];
arr[0] = subResult[0];
arr[1] = subResult[2];
arr[2] = subResult[1] - (a/b) * subResult[2];
return arr;
}
public int secureRandom(int max) {
return rand.nextInt(max);
}
public BigInteger getGenerator(BigInteger p, BigInteger q){
BigInteger generator = BigInteger.ZERO;
BigInteger exp = p.subtract(BigInteger.ONE).divide(q);
for(int i=2;i<p.intValue()-1;i++) {
generator = BigInteger.valueOf(i);
generator = generator.modPow(exp, p);
if(!generator.equals(BigInteger.ONE)) return generator;
}
// do {
// generator = BigInteger.valueOf(this.secureRandom(p.intValue()));
// generator = generator.modPow(exp, p);
// } while(generator.equals(BigInteger.ONE));
return generator;
// BigInteger generator = BigInteger.ZERO;
// for(long i = 2; i < p.intValue()-1; i++)
// {
// generator = BigInteger.valueOf(i);
// if(generator.modPow(BigInteger.valueOf(2), p).equals(BigInteger.ONE)) continue;
// if(generator.modPow(q, p).equals(BigInteger.ONE)) continue;
// else return generator;
// }
// return generator;
}
public String readMessageFile(String filename) {
try {
return new String(Files.readAllBytes(Paths.get(filename)));
} catch(Exception ex) {
System.out.println("ERR: Can't read file.");
}
return null;
}
public int[] readSignatureFile(String filename) {
InputStream is=null;
DataInputStream dis=null;
try {
is = new FileInputStream(filename);
dis = new DataInputStream(is);
ArrayList intList = new ArrayList();
int count = 0;
while(dis.available()>0) {
intList.add(dis.readInt());
}
int[] data = new int[intList.size()];
for(int i=0;i<data.length;i++) {
data[i] = (int)intList.get(i);
}
dis.close();
is.close();
return data;
} catch(Exception ex) {
System.out.println("ERR: Can't read file.");
}
return null;
}
public static void writeSignatureFile(String filename, int[] contents) {
FileOutputStream fos=null;
DataOutputStream dos=null;
try {
fos = new FileOutputStream(filename);
dos = new DataOutputStream(fos);
for(int i=0;i<contents.length;i++) {
dos.writeInt(contents[i]);
}
dos.flush();
dos.close();
fos.close();
} catch(Exception ex) {
System.out.println("ERR: Can't write to file.");
}
}
public void keyGeneration() {
System.out.println("UNDENIABLE SIGNATURE SCHEME");
System.out.println("- KEY GENERATION");
System.out.print("\tEnter prime P (such that P=2Q+1) : ");
this.P = new Scanner(System.in).nextInt();
this.Q = (this.P - 1)/2;
System.out.println("\tPrime Q is "+this.Q+".");
this.G = this.getGenerator(BigInteger.valueOf(this.P),BigInteger.valueOf(this.Q));
System.out.println("\tGenerator G of group Z*("+this.P+") is "+this.G+".");
System.out.print("\tEnter private key D (belongs to Z*P) :");
this.D = new Scanner(System.in).nextInt();
while(this.D <= 0 || this.D >= this.P) {
System.out.println("\tInvalid private key selected, please try again.");
System.out.print("\tEnter private key D (belongs to Z*P) :");
this.D = new Scanner(System.in).nextInt();
}
this.Y = this.G.modPow(BigInteger.valueOf(this.D), BigInteger.valueOf(this.P));
System.out.println("\tG^D(mod P) = "+this.Y+".");
System.out.println("\tPublic Key [P, G, Y] is ["+this.P+", "+this.G+", "+this.Y+"].");
System.out.println("\tPrivate Key [D] is ["+this.D+"].");
int[] g = this.extendedEuclidean(this.D, this.Q);
this.invD = g[1];
if(this.invD < 0) {
this.invD = this.Q - Math.abs(this.invD);
System.out.println("\tInverse was negative.");
}
System.out.println("\tInverse of D (mod Q) is "+this.invD+".");
}
public void messageSigning(String message) {
int[] signatureBytes = new int[message.length()];
for(int i=0;i<message.length();i++) {
BigInteger m = BigInteger.valueOf((int)message.charAt(i));//this.H(message.charAt(i)+"");
BigInteger s = m.pow(this.D).mod(BigInteger.valueOf(this.P));
signatureBytes[i] = s.intValue();
// System.out.println("\tchar:"+message.charAt(i)+", m:"+m+", s:"+s+", S:"+signatureBytes[i]);
}
System.out.print("\tEnter signature filename : ");
String signatureFilename = new Scanner(System.in).nextLine();
this.writeSignatureFile(signatureFilename, signatureBytes);
System.out.println("\tSignature is successfully generated.");
}
// see the condition proof for w=W for solution of problem
public boolean signatureVerification(String message, int[] signature) {
for(int i=0;i<message.length();i++) {
// rand=new Random();
// int x1 = this.secureRandom(this.Q-1);
// while(x1<=1) x1 = this.secureRandom(this.Q-1);
// int x2 = this.secureRandom(this.Q-1);
// while(x2<=1) x2 = this.secureRandom(this.Q-1);
int x1 = 38, x2 = 397;
BigInteger S = BigInteger.valueOf(signature[i]);
BigInteger s = S.pow(x1);
BigInteger y = this.Y.pow(x2);
BigInteger z = s.multiply(y).mod(BigInteger.valueOf(this.P));
BigInteger w = signerChallengeForVerification(z);
BigInteger M = BigInteger.valueOf((int)message.charAt(i));//this.H(message.charAt(i)+"");
BigInteger m = M.pow(x1);
BigInteger g = this.G.pow(x2);
BigInteger mg = m.multiply(g);
BigInteger W = mg.mod(BigInteger.valueOf(this.P));
// System.out.println("\ti:"+i+", char:"+message.charAt(i)+", m:"+M+", s:"+S);
// System.out.println("\t\tz:"+z+", w:"+w+" , W:"+W);
if(!w.equals(W)) {
return false;
}
}
return true;
}
public boolean disavowalProtocol(String message, int[] signature) {
int x1 = 38, x2 = 397;
BigInteger response1 = BigInteger.ZERO;
BigInteger response2 = BigInteger.ZERO;
// verification 1
for(int i=0;i<message.length();i++) {
BigInteger S = BigInteger.valueOf(signature[i]);
BigInteger s = S.pow(x1);
BigInteger y = this.Y.pow(x2);
BigInteger z = s.multiply(y).mod(BigInteger.valueOf(this.P));
BigInteger w = signerChallengeForVerification(z);
BigInteger M = BigInteger.valueOf((int)message.charAt(i));//this.H(message.charAt(i)+"");
BigInteger m = M.pow(x1);
BigInteger g = this.G.pow(x2);
BigInteger mg = m.multiply(g);
BigInteger W = mg.mod(BigInteger.valueOf(this.P));
// System.out.println("\ti:"+i+", char:"+message.charAt(i)+", m:"+M+", s:"+S);
// System.out.println("\t\tz:"+z+", w:"+w+" , W:"+W);
if(!w.equals(W)) {
response1 = w;
break;
}
}
if(response1.equals(BigInteger.ZERO)) {
return false; // signature is not forgery bcoz signature is matched
}
// verification 2
for(int i=0;i<message.length();i++) {
BigInteger S = BigInteger.valueOf(signature[i]);
BigInteger s = S.pow(x1);
BigInteger y = this.Y.pow(x2);
BigInteger z = s.multiply(y).mod(BigInteger.valueOf(this.P));
BigInteger w = signerChallengeForVerification(z);
BigInteger M = BigInteger.valueOf((int)message.charAt(i));//this.H(message.charAt(i)+"");
BigInteger m = M.pow(x1);
BigInteger g = this.G.pow(x2);
BigInteger mg = m.multiply(g);
BigInteger W = mg.mod(BigInteger.valueOf(this.P));
// System.out.println("\ti:"+i+", char:"+message.charAt(i)+", m:"+M+", s:"+S);
// System.out.println("\t\tz:"+z+", w:"+w+" , W:"+W);
if(!w.equals(W)) {
response2 = w;
break;
}
}
if(response2.equals(BigInteger.ZERO)) {
return false; // signature is not forgery bcoz signature is matched
}
// forgery validation now it comes here only when one of above validation is failed
BigInteger A = response1.divide(this.G.pow(x2)).pow(x1).mod(BigInteger.valueOf(this.P));
BigInteger B = response2.divide(this.G.pow(x1)).pow(x2).mod(BigInteger.valueOf(this.P));
return A.equals(B); // signature is forgery if A == B
}
public BigInteger signerChallengeForVerification(BigInteger z) {
// System.out.println("\tchallange: "+z+", invD:"+this.invD+", d:"+(z.pow(this.invD).mod(BigInteger.valueOf(this.P))));
return z.pow(this.invD).mod(BigInteger.valueOf(this.P));
}
}
class Program {
public static void main(String[] args) {
String msgFilename, signatureFilename, message;
UndeniableSignature scheme = new UndeniableSignature();
scheme.keyGeneration();
int choice = 0;
do {
System.out.println();
System.out.println("- OPERATIONS");
System.out.println("\t1. Sign message");
System.out.println("\t2. Verify signature");
System.out.println("\t3. Disavowal protocal");
System.out.println("\t4. Exit");
System.out.print("\tChoice : ");
choice = new Scanner(System.in).nextInt();
System.out.println();
switch(choice) {
case 1:
System.out.println("- MESSAGE SIGNING");
System.out.print("\tEnter message filename : ");
msgFilename = new Scanner(System.in).nextLine();
message = new String(scheme.readMessageFile(msgFilename));
scheme.messageSigning(message);
break;
case 2:
System.out.println("- SIGNATURE VERIFICATION");
System.out.print("\tEnter message filename : ");
msgFilename = new Scanner(System.in).nextLine();
System.out.print("\tEnter signature filename : ");
signatureFilename = new Scanner(System.in).nextLine();
message = scheme.readMessageFile(msgFilename);
int[] signatureBytes = scheme.readSignatureFile(signatureFilename);
if(scheme.signatureVerification(message, signatureBytes)) {
System.out.println("\tSignature IS valid.");
} else {
System.out.println("\tSignature is NOT valid.");
}
break;
case 3:
System.out.println("- DISAVOWAL VERIFICATION");
System.out.print("\tEnter message filename : ");
msgFilename = new Scanner(System.in).nextLine();
System.out.print("\tEnter signature filename : ");
signatureFilename = new Scanner(System.in).nextLine();
message = scheme.readMessageFile(msgFilename);
signatureBytes = scheme.readSignatureFile(signatureFilename);
if(scheme.disavowalProtocol(message, signatureBytes)) {
System.out.println("\tSignature IS forgery.");
} else {
System.out.println("\tSignature is NOT forgery.");
}
break;
case 4:
System.out.println("- BYE BYE");
break;
}
} while(choice!=4);
}
}
thanks :D
i have followed https://gist.github.com/xrstf/b48a970098a8e76943b9 to integrate nutch and elastic-search. everything is working fine data is stored in Hbase 'webpage' table but i am not able to fetch data in elastic search.i want to know how to fetch data in elastic search.
below is my code
package com.process;
/*
import package will be here
*/
public class HbaseToElastic extends Configured implements
org.apache.hadoop.util.Tool {
static class Mapper extends TableMapper<Text, IndexWritable> {
public static String CLUSTER;
public static String SEARCH_HOST;
public static String SEARCH_PORT;
public static String SEARCH_INDEX_NAME;
public static String SEARCHtYPE;
public static int BULKSIZE;
public static String TABLENAME;
public static String FAMILY;
private static List<String> SPORTS_KEYWORDS;
private static List<String> BUSINESS_KEYWORDS;
private static List<String> GOSSIP_KEYWORDS;
private static List<String> CRIME_KEYWORDS;
private static Map<String, Map<String, String>> STATE_MAP = new HashMap<String, Map<String, String>>();
private static Map<String, String> CITY_MAP = new HashMap<String, String>();
private static Mapper mapper = new Mapper();
static {
try {
System.out.println("done1");
DetectorFactory.loadProfile("./profiles");
System.out.println("done2");
} catch (final LangDetectException e) {
System.out.println("done3");
e.printStackTrace();
}
}
Configuration hbaseConf = null;
HTable table = null;
List<Put> hbasePutErrorList = new ArrayList<Put>();
/**
* Clean up the hbase table object
*/
#Override
protected void cleanup(final Context context) throws IOException,
InterruptedException {
super.cleanup(context);
table.put(hbasePutErrorList);
table.close();
hbasePutErrorList.clear();
}
/**
* Initialize various variables
*/
#Override
protected void setup(
final org.apache.hadoop.mapreduce.Mapper<ImmutableBytesWritable, Result, Text, IndexWritable>.Context context)
throws IOException, InterruptedException {
final Configuration conf = context.getConfiguration();
CLUSTER = conf.get("cluster");
SEARCH_HOST = conf.get("search_host");
SEARCH_PORT = conf.get("search_port");
SEARCH_INDEX_NAME = conf.get("search_index_name");
SEARCHtYPE = conf.get("search_type");
BULKSIZE = conf.getInt("search_bulk_size", 500);
TABLENAME = conf.get("table_name");
FAMILY = conf.get("family");
hbaseConf = HBaseConfiguration.create();
hbaseConf.set("hbase.zookeeper.quorum",
conf.get("hbase.zookeeper.quorum"));
hbaseConf.set("hbase.zookeeper.property.clientPort",
conf.get("hbase.zookeeper.property.clientPort"));
hbaseConf.set("hbase.rpc.timeout", conf.get("hbase.rpc.timeout"));
hbaseConf.set("hbase.regionserver.lease.period",
conf.get("hbase.regionserver.lease.period"));
hbaseConf.set("hbase.master", conf.get("hbase.master"));
table = new HTable(hbaseConf, conf.get("table_name"));
SPORTS_KEYWORDS = new ArrayList<String>();
BUSINESS_KEYWORDS = new ArrayList<String>();
GOSSIP_KEYWORDS = new ArrayList<String>();
CRIME_KEYWORDS = new ArrayList<String>();
String keywrods = conf.get("sportskeywords");
String[] keyarr = keywrods.split(",");
for (final String key : keyarr) {
SPORTS_KEYWORDS.add(key.trim());
}
keywrods = conf.get("businesskeywords");
keyarr = keywrods.split(",");
for (final String key : keyarr) {
BUSINESS_KEYWORDS.add(key.trim());
}
keywrods = conf.get("gossipkeywords");
keyarr = keywrods.split(",");
for (final String key : keyarr) {
GOSSIP_KEYWORDS.add(key.trim());
}
keywrods = conf.get("crimekeywords");
keyarr = keywrods.split(",");
for (final String key : keyarr) {
CRIME_KEYWORDS.add(key.trim());
}
final String stateMap = conf.get("statemap");
final Gson g = new Gson();
STATE_MAP = g.fromJson(stateMap, Map.class);
}
/**
* map function
*/
#Override
public void map(final ImmutableBytesWritable row, final Result result,
final Context context) throws IOException, InterruptedException {
try {
final byte b = 0;
int deleteFlag = 0;
final String keyString = Bytes.toString(row.get());
final Map<String, Object> mapobject = new HashMap<String, Object>();
for (final KeyValue kv : result.raw()) {
final String key = (new String(kv.getQualifier()));
final String value = (new String(kv.getValue()));
mapobject.put(key, value);
}
final Gson g = new Gson();
if (checkValidType(mapobject)) {
refineMetaTags(mapobject);
if (refineDescription(mapobject)) {
assignCity(mapobject);
if (checkTitleImage(mapobject)) {
if (setLang(mapobject)) {
setCorrectCategory(mapobject);
correctDuplicateTitle(mapobject);
final String json = g.toJson(mapobject);
context.write(new Text(keyString),
new IndexWritable(json, b));
deleteFlag = 1;
}
}
}
}
if (deleteFlag == 0) {
final Put put = new Put(Bytes.toBytes(keyString));
put.add(Bytes.toBytes("cf"), Bytes.toBytes("ErrorFlag"),
Bytes.toBytes("1"));
hbasePutErrorList.add(put);
}
} catch (final Exception e) {
e.printStackTrace();
}
}
/**
* Remove duplicate statement in the title
*
* #param mapobject
*/
private void correctDuplicateTitle(final Map<String, Object> mapobject) {
final String duplicateTitle = mapobject.get("title").toString();
final String stripedTitleArr[] = duplicateTitle.split(" ", 4);
if (stripedTitleArr.length == 4) {
final String subString = stripedTitleArr[0] + " "
+ stripedTitleArr[1] + " " + stripedTitleArr[2];
if (stripedTitleArr[3].contains(subString)) {
mapobject.put("title", duplicateTitle
.substring(duplicateTitle.indexOf(subString,
subString.length() - 1)));
mapobject.put("title", stripedTitleArr[3]
.substring(stripedTitleArr[3].indexOf(subString)));
}
}
}
/**
* Set category based on the various category specific keyword
*
* #param mapobject
*/
private void setCorrectCategory(final Map<String, Object> mapobject) {
final String url = mapobject.get("url") + "";
final String cat = mapobject.get("tags") + "";
if ("sports".equalsIgnoreCase(cat)
|| "cricket".equalsIgnoreCase(cat)) {
if (!(url.toLowerCase().contains("sport")
|| url.toLowerCase().contains("खेल")
|| url.toLowerCase().contains("cric") || url
.toLowerCase().contains("क्रिकेट"))) {
final String desc = mapobject.get("description").toString();
boolean isSports = false;
int count = 0;
for (final String keyword : SPORTS_KEYWORDS) {
if (desc.contains(keyword)) {
count++;
}
}
if (count > 1) {
isSports = true;
}
if (!isSports) {
mapobject.put("tags", "national");
}
if (isSports
&& (desc.contains("क्रिकेट")
|| url.toLowerCase().contains("cric")
|| desc.contains("टॉस")
|| desc.contains("वनडे") || desc
.contains("बल्लेबाज"))) {
mapobject.put("tags", "cricket");
}
}
} else if ("business".equalsIgnoreCase(cat)) {
if ((url.toLowerCase().contains("sport") || url.toLowerCase()
.contains("खेल"))) {
mapobject.put("tags", "sports");
} else if (url.toLowerCase().contains("cric")
|| url.toLowerCase().contains("क्रिकेट")) {
mapobject.put("tags", "cricket");
} else if (!(url.toLowerCase().contains("busines")
|| url.toLowerCase().contains("व्यापार")
|| url.toLowerCase().contains("economy")
|| url.toLowerCase().contains("finance")
|| url.toLowerCase().contains("बिजनेस")
|| url.toLowerCase().contains("market")
|| url.toLowerCase().contains("karobar") || url
.contains("कारोबार"))) {
final String desc = mapobject.get("description").toString();
int count = 0;
for (final String keyword : BUSINESS_KEYWORDS) {
if (desc.contains(keyword)) {
count++;
}
}
if (count < 2) {
mapobject.put("tags", "national");
}
}
} else if ("gossip".equalsIgnoreCase(cat)) {
if ((url.toLowerCase().contains("sport") || url.toLowerCase()
.contains("खेल"))) {
mapobject.put("tags", "sports");
} else if (url.toLowerCase().contains("cric")
|| url.toLowerCase().contains("क्रिकेट")) {
mapobject.put("tags", "cricket");
} else if (url.toLowerCase().contains("busines")) {
mapobject.put("tags", "business");
} else if (!(url.toLowerCase().contains("masala")
|| url.toLowerCase().contains("gossip")
|| url.toLowerCase().contains("gupshup") || url
.toLowerCase().contains("garam"))) {
final String desc = mapobject.get("description").toString();
int count = 0;
for (final String keyword : GOSSIP_KEYWORDS) {
if (desc.contains(keyword)) {
count++;
}
}
if (count < 2) {
mapobject.put("tags", "national");
}
}
} else if ("crime".equalsIgnoreCase(cat)) {
if ((url.toLowerCase().contains("sport") || url.toLowerCase()
.contains("खेल"))) {
mapobject.put("tags", "sports");
} else if (url.toLowerCase().contains("cric")
|| url.toLowerCase().contains("क्रिकेट")) {
mapobject.put("tags", "cricket");
} else if (url.toLowerCase().contains("busines")) {
mapobject.put("tags", "business");
} else if (!(url.toLowerCase().contains("crime")
|| url.toLowerCase().contains("terrorist")
|| url.toLowerCase().contains("abuse")
|| url.toLowerCase().contains("forgery")
|| url.toLowerCase().contains("assault")
|| url.toLowerCase().contains("violence")
|| url.toLowerCase().contains("rape")
|| url.toLowerCase().contains("teasing")
|| url.toLowerCase().contains("molestation")
|| url.toLowerCase().contains("scandal") || url
.toLowerCase().contains("murder"))) {
final String desc = mapobject.get("description").toString();
int count = 0;
for (final String keyword : CRIME_KEYWORDS) {
if (desc.contains(keyword)) {
count++;
}
}
if (count < 2) {
mapobject.put("tags", "national");
}
}
} else if (cat != null && cat.startsWith("local")) {
}
}
/**
* Check valid type of the HTML pages
*
* #param mapobject
* #return
*/
private boolean checkValidType(final Map<String, Object> mapobject) {
if (mapobject.containsKey("type")
&& !(mapobject.get("type").toString().contains("image") || mapobject
.get("type").toString().contains("rss"))) {
return true;
}
return false;
}
/**
* refine the description according to its length and must starting with
* english and it the description is not present get the description
* from the metatags description
*
* #param mapobject
* #return {#link Boolean}
*/
private boolean refineDescription(final Map<String, Object> mapobject) {
if (mapobject.containsKey("description")
&& mapobject.get("description").toString().length() > 75
&& !mapobject.get("description").toString().contains(";}")
&& !mapobject.get("description").toString()
.contains("<cite>")
&& !mapobject.get("description").toString()
.contains("href=")
&& !mapobject.get("description").toString()
.contains("All rights reserved")) {
return true;
} else if (mapobject.containsKey("metatag.description")
&& mapobject.get("metatag.description").toString().length() > 75
&& !mapobject.get("metatag.description").toString()
.contains(";}")
&& !mapobject.get("metatag.description").toString()
.contains("<cite>")) {
mapobject.put("description",
mapobject.get("metatag.description"));
return true;
}
return false;
}
/**
* refine metatags by refining meta keyword to only include the English
* keyword only that has at most three keyword and if not present then
* create the keyword with title field of the html and if none of the
* keyword found then form it using the help of the url and exclude the
* number from the keywords
*
* #param mapobject
*/
private void refineMetaTags(final Map<String, Object> mapobject) {
String metaTag = "";
int tagFlag = 0;
if (mapobject.containsKey("metatag.keywords")) {
final String metaTags[] = mapobject.get("metatag.keywords")
.toString().replaceAll("\\|", ",").split(",");
String domain = null;
StringBuilder temp = null;
for (final String metaTag2 : metaTags) {
if (mapobject.containsKey("host")) {
domain = mapobject.get("host") + "";
if (domain.split("\\.").length > 1
&& (metaTag2
.contains(domain.split("\\.")[domain
.split("\\.").length - 2]) || metaTag2
.contains(domain.split("\\.")[0])))
{
continue;
}
}
String[] arr = metaTag2.split(" ");
arr = removeUnicodeWords(arr);
if (arr.length > 0 && arr.length < 5) {
temp = new StringBuilder();
for (final String str : arr) {
temp.append(str);
temp.append(" ");
}
if (metaTag.length() + temp.length() < 70) {
metaTag = metaTag + "," + temp.toString();
}
}
}
if (metaTag.startsWith(",")) {
metaTag = metaTag.trim();
metaTag = metaTag.substring(1, metaTag.length());
}
}
if (metaTag.length() < 1 && mapobject.containsKey("title")) {
/**
* Extracting tags from the title tag if the length of the
* keyword is greater than 4
*/
final String title = (String) mapobject.get("title");
final String splitTitle[] = title.split(" ");
int count = 0;
for (int i = 0; i < splitTitle.length; i++) {
if (splitTitle[i].length() > 4
&& !splitTitle[i].matches("^[\\u0900-\\u097F].*")) {
metaTag = metaTag + splitTitle[i] + ",";
count++;
if (count == 5) {
break;
}
}
}
if (metaTag.split(",").length > 3) {
if (metaTag.endsWith(",")) {
metaTag = metaTag.trim();
metaTag = metaTag.substring(0, metaTag.length() - 1);
}
} else {
metaTag = "";
}
}
if (metaTag.length() < 1) {
/**
* Extracting the tags from the url if the length of the keyword
* is greater than 4
*/
final String splitUrl[] = mapobject.get("url").toString()
.split("/");
final String lastSplitValue = splitUrl[splitUrl.length - 1];
final String tagList[] = generateTokens(lastSplitValue);
if (tagList != null) {
int count = 0;
for (int i = 0; i < tagList.length; i++) {
if (tagList[i].length() > 4
&& !tagList[i].matches("^[\\u0900-\\u097F].*")) {
metaTag = metaTag + tagList[i] + ",";
count++;
if (count == 5) {
break;
}
}
}
}
if (metaTag.endsWith(",")) {
metaTag = metaTag.trim();
metaTag = metaTag.substring(0, metaTag.length() - 1);
}
}
if (metaTag.length() > 0) {
metaTag = metaTag.replaceAll("\\[", "");
metaTag = metaTag.replaceAll("\"", "");
metaTag = metaTag.replaceAll(";", "");
metaTag = metaTag.replaceAll(":", "");
metaTag = metaTag.replaceAll("\u0027", "");
metaTag = metaTag.replaceAll("\u003d", "");
metaTag = metaTag.replaceAll("\u0026", "");
tagFlag = 1;
}
mapobject.put("TagFlag", tagFlag);
mapobject.put("metatag.keywords", metaTag);
}
/**
* Remove unicode character
*
* #param arr
* #return
*/
private String[] removeUnicodeWords(final String[] arr) {
final List<String> returnArr = new ArrayList<String>();
for (final String str : arr) {
if (str != null && str.trim().length() > 3
&& !str.matches("^[\\u0900-\\u097F].*")
&& !(str.matches("^[0-9].*"))) {
returnArr.add(str.trim());
}
}
final String[] retrnArr = new String[returnArr.size()];
returnArr.toArray(retrnArr);
return retrnArr;
}
/**
* Generate Token list with the help of the lucene analyzer
*
* #param lastSplitValue
* #return {#link ArrayIndexOutOfBoundsException} of the list of the
* keywords
*/
private String[] generateTokens(String lastSplitValue) {
final List<String> list = new ArrayList<String>();
lastSplitValue = lastSplitValue.replace("\\.", " ").replace("%20",
" ");
try {
final Version matchVersion = Version.LUCENE_45;
final Analyzer analyzer = new HindiAnalyzer(matchVersion);
final TokenStream ts = analyzer.tokenStream("field",
new StringReader(lastSplitValue));
ts.reset();
while (ts.incrementToken()) {
final CharTermAttribute cta = ts
.getAttribute(CharTermAttribute.class);
if (cta.toString().length() > 4
&& !cta.toString().matches("^[0-9].*")) {
list.add(cta.toString());
}
}
ts.end();
ts.close();
analyzer.close();
} catch (final Exception e) {
e.printStackTrace();
}
if (list.size() > 3) {
return list.toArray(new String[list.size()]);
} else {
return null;
}
}
/**
* Checks title and assign their language based on their first character
* of the title
*
* #param mapobject
* #return {#link Map}
*/
private boolean setLang(final Map<String, Object> mapobject) {
final String title = mapobject.get("title").toString();
final String description = mapobject.get("title").toString();
String language = "";
try {
language = mapper.detect(title);
mapper.detect(description);
} catch (final LangDetectException e) {
System.out.println("\n title with error is - " + title);
System.out.println("\n description with error is - "
+ description);
e.printStackTrace();
/*
* String title = mapobject.get("title").toString(); language =
* mapobject.get("lang") + ""; language = language.trim(); if
* (language.trim().equalsIgnoreCase("hi") ||
* language.trim().startsWith("en") ||
* language.trim().equalsIgnoreCase("lt")) { String[] titleArr =
* title.trim().split(" "); int i = 0; for (String titlePart :
* titleArr) { if
* (titlePart.trim().matches("^[\\u0900-\\u097F].*")) { i++; } }
* if (i >= titleArr.length * 0.5) { mapobject.put("lang",
* "hi"); } else { mapobject.put("lang", "lt"); } return true; }
*/
return false;
}
if (language.trim().equalsIgnoreCase("hi")
|| language.trim().startsWith("en")
|| language.trim().equalsIgnoreCase("lt")) {
mapobject.put("lang", language);
return true;
}
return false;
}
private String detect(final String text) throws LangDetectException {
final Detector detector = DetectorFactory.create();
detector.append(text);
return detector.detect();
}
/**
* Checks whether to include the doc based on their title and get the
* title from anchor tag title to choose the title that has largest
* number of the words and in hindi and it also gets the image from
* anchor tag href attribute
*
* #param mapobject
* of the key value pair
* #return {#link Boolean}
*/
private boolean checkTitleImage(final Map<String, Object> mapobject) {
final TreeSet<String> set = new TreeSet<String>(new SetSort());
final Gson gson = new Gson();
JsonArray array = null;
JsonObject object2 = null;
if (mapobject.containsKey("anchor")
&& mapobject.get("anchor") != null) {
final String arr = (String) mapobject.get("anchor");
try {
array = gson.fromJson(arr, JsonArray.class);
for (final JsonElement jsonElement : array) {
try {
object2 = gson.fromJson(jsonElement.getAsString(),
JsonObject.class);
} catch (final Exception e) {
if (object2 == null) {
object2 = new JsonObject();
object2.addProperty("title",
jsonElement.getAsString());
object2.addProperty("href", "");
object2.addProperty("alt", "");
}
}
if (object2 != null) {
assignTitleImage(mapobject, set, object2);
}
object2 = null;
}
} catch (final ClassCastException e) {
object2 = gson.fromJson(arr, JsonObject.class);
assignTitleImage(mapobject, set, object2);
} catch (final Exception e) {
e.printStackTrace();
}
if (!set.isEmpty()) {
int loop = 0;
final List<String> tempList = new LinkedList<String>();
for (final String string : set) {
final String title = string;
tempList.add(title.trim());
loop++;
if (loop == 2) {
break;
}
}
if (!tempList.isEmpty()) {
if (tempList.get(0).matches("^[\\u0900-\\u097F].*")) {
mapobject.put("title", tempList.get(0));
} else if (tempList.size() > 1
&& !(tempList.get(0)
.matches("^[\\u0900-\\u097F].*"))
&& tempList.get(1).matches(
"^[\\u0900-\\u097F].*")) {
mapobject.put("title", tempList.get(1));
} else {
mapobject.put("title", tempList.get(0));
}
}
}
}
if (mapobject.containsKey("title")
&& mapobject.get("title").toString().length() > 0
&& mapobject.get("title").toString().split(" ").length > 2
&& mapobject.get("title").toString().split(" ").length < 20
&& !mapobject.get("title").toString().contains("<")) {
if (set.isEmpty()) {
mapobject.put("title",
getTitleRefined(mapobject.get("title") + ""));
}
return true;
}
return false;
}
/**
* #param mapobject
* #param set
* #param object2
*/
private void assignTitleImage(final Map<String, Object> mapobject,
final TreeSet<String> set, final JsonObject object2) {
if (!mapobject.containsKey("ImgH1")
&& !mapobject.containsKey("ImgH2")) {
if (object2.get("href") != null
&& object2.get("href").getAsString().length() > 0
&& (object2.get("href").getAsString().toLowerCase()
.contains(".jpg")
|| object2.get("href").getAsString()
.toLowerCase().contains(".jpeg") || object2
.get("href").getAsString().toLowerCase()
.contains(".gif"))) {
putImages(mapobject, object2.get("href").getAsString()
.trim(), mapobject.get("tags").toString().trim()
.toLowerCase());
}
}
if (object2.get("title") != null
&& object2.get("title").getAsString().length() > 0
&& object2.get("title").getAsString().split(" ").length > 2
&& object2.get("title").getAsString().split(" ").length < 20
&& !object2.get("title").getAsString().contains("<")) {
final String newTitle = getTitleRefined(object2.get("title")
.getAsString());
set.add(newTitle.trim());
}
}
/**
* This function used to refine the title based on specific bad keyword
* during observation
*
* #param title
* #return refined title
*/
private String getTitleRefined(String title) {
title = title.replaceAll("\u0027", "");
title = title.replaceAll("\u0026", "");
title = title.replaceAll("\u003d", "");
if (title.contains("-")) {
if (title.trim().split("-").length > 1
&& !title.trim().split("-")[1].trim().matches(
"^[\\u0900-\\u097F].*")) {
return title.trim().split("-")[0].trim();
}
} else if (title.contains(":")) {
if (!title.trim().split(":")[0].trim().matches(
"^[\\u0900-\\u097F].*")
&& title.trim().split(":").length > 1) {
return title.trim().split(":")[1].trim();
}
}
return title;
}
/**
* Creates the path for the images
*
* #param map
* of the key value pair
* #param imageUrl
* #param category
*/
private void putImages(final Map<String, Object> map2,
final String imageUrl, final String category) {
try {
map2.put("ImgSrc", StringEscapeUtils.unescapeHtml(imageUrl)
.trim());
if (map2.containsKey("ImgSrc") && map2.get("ImgSrc") != null
&& map2.get("ImgSrc").toString().length() > 0) {
map2.put(
"ImgSrc",
StringEscapeUtils.unescapeHtml(map2.get("ImgSrc")
.toString())
+ "##RAFTAAR##"
+ imageUrl.trim());
} else {
return;
}
String imgNamearr[] = null;
try {
imgNamearr = imageUrl.split("/");
} catch (final Exception e) {
e.printStackTrace();
}
String imgName = null;
try {
imgName = imgNamearr[imgNamearr.length - 1];
} catch (final Exception e) {
e.printStackTrace();
}
final String imagePath = "/"
+ String.valueOf(imgName.charAt(0));
imgName = imgName.replaceAll(" ", "_").replaceAll("%20", "_");
if (imgName.split(".jpg").length > 0) {
imgName = imgName.split(".jpg")[0];
imgName = imgName + ".jpg";
}
map2.put("ImgH1", "h1/" + category + imagePath + "/" + imgName);
map2.put("ImgH2", "h2/" + category + imagePath + "/" + imgName);
} catch (final Exception e) {
e.printStackTrace();
}
}
/**
* Inserts the data to the elasticsearch
*
* #param mapobject
* #param key
* unique id generally it is the unique url
*/
public static void insertToElastic(final Map<String, Object> mapobject,
final String key) {
final Settings settings = ImmutableSettings.settingsBuilder()
.put("cluster.name", CLUSTER).build();/*
* change ccluster.name
* to cluster
*/
final Client client = new TransportClient(settings)
.addTransportAddress(new InetSocketTransportAddress(
SEARCH_HOST, Integer.parseInt(SEARCH_PORT)));
client.prepareIndex(SEARCH_INDEX_NAME, SEARCHtYPE, key)
.setSource(mapobject).execute().actionGet();
client.close();
}
/**
* Assign the city to the news without city
*
* #param category
* #param description
* #return update category with city
*/
private static void assignCity(final Map<String, Object> mapobject) {
String category = mapobject.get("tags").toString();
if (category.endsWith("/")) {
boolean flag = true;
final String catArr[] = category.split("/");
if (catArr.length == 2) {
final String state = catArr[1];
CITY_MAP = STATE_MAP.get(state);
for (final Entry<String, String> e : CITY_MAP.entrySet()) {
final String description = mapobject.get("description")
.toString();
if (description.contains(e.getValue())) {
category = category + e.getKey();
mapobject.put("tags", category);
flag = false;
break;
}
}
}
if (flag) {
mapobject.put("tags", "national");
}
}
}
}
/**
* Update the data to hbase
*
* #param tableName
* #param rowKey
* #param family
* #param qualifier
* #param value
* #param conf
*/
public static void updateIntoHbase(final String tableName,
final String rowKey, final String family, final String qualifier,
final String value, final Configuration conf) {
HTable table = null;
try {
table = new HTable(conf, tableName);
} catch (final IOException e) {
e.printStackTrace();
}
final Put put = new Put(Bytes.toBytes(rowKey));
put.add(Bytes.toBytes(family), Bytes.toBytes(qualifier),
Bytes.toBytes(value));
try {
table.put(put);
table.close();
} catch (final IOException e) {
e.printStackTrace();
}
}
/**
* Return the map of the all states and city
*
* #param fileName
* #return
*/
private static Map<String, Map<String, String>> returnMap(
final String fileName) {
final Map<String, Map<String, String>> map = new HashMap<String, Map<String, String>>();
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader(fileName));
String line;
while ((line = br.readLine()) != null) {
final String arr[] = line.split("\t", 3);
if (arr.length == 3) {
if (map.containsKey(arr[0])) {
Map<String, String> m = new HashMap<String, String>();
m = map.get(arr[0]);
m.put(arr[1], arr[2]);
} else {
final Map<String, String> m = new HashMap<String, String>();
m.put(arr[1], arr[2]);
map.put(arr[0], m);
}
}
}
} catch (final FileNotFoundException e) {
e.printStackTrace();
} catch (final IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (final Exception e) {
e.printStackTrace();
}
}
}
return map;
}
public static void main(final String[] args) throws Exception {
int c = 0;
c = ToolRunner.run(new Configuration(), new HbaseToElastic(), args);
System.exit(c);
}
}
I am a newbie of Blackberry developing application. I try to store all xml parsing data to an object, and set them to a vector.
public class XmlParser extends MainScreen {
Database d;
private HttpConnection hcon = null;
private Vector binN;
public Vector getBinN() {
return binN;
}
public void setBinN(Vector bin) {
this.binN = bin;
}
LabelField from;
LabelField ttl;
LabelField desc;
LabelField date;
public XmlParser() {
LabelField title = new LabelField("Headline News" ,LabelField.HCENTER|LabelField.USE_ALL_WIDTH);
setTitle(title);
try {
URI myURI = URI.create("file:///SDCard/Database/WebFeed.db");
d = DatabaseFactory.open(myURI);
Statement st = d.createStatement("SELECT feed_url, feed_name FROM WebFeed");
st.prepare();
Cursor c = st.getCursor();
while (c.next()) {
Row r = c.getRow();
hcon = (HttpConnection)Connector.open(r.getString(0));
hcon.setRequestMethod(HttpConnection.GET);
hcon.setRequestProperty("User-Agent", "Profile/MIDP-1.0 Configuration/CLDC-1.0");
hcon.setRequestProperty("Content-Length", "0");
hcon.setRequestProperty("Connection", "close");
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
builder.isValidating();
Document document = builder.parse(hcon.openInputStream());
Element rootElement = document.getDocumentElement();
rootElement.normalize();
NodeList list = document.getElementsByTagName("item");
int i=0;
while (i<10){
Node item = list.item(i);
if(item.getNodeType() != Node.TEXT_NODE) {
NodeList itemChilds = item.getChildNodes();
int j=0;
while (j<10){
Node detailNode = itemChilds.item(j);
if(detailNode.getNodeType() != Node.TEXT_NODE) {
if(detailNode.getNodeName().equalsIgnoreCase("title")) {
ttl = new LabelField(getNodeValue(detailNode)) {
public void paint(Graphics g) {
g.setColor(Color.BLUE);
super.paint(g);
}
};
from = new LabelField(r.getString(1), LabelField.FIELD_RIGHT|LabelField.USE_ALL_WIDTH);
ttl.setFont(Font.getDefault().derive(Font.BOLD));
from.setFont(Font.getDefault().derive(Font.BOLD));
add (from);
add (ttl);
} else if(detailNode.getNodeName().equalsIgnoreCase("description")) {
desc = new LabelField(getNodeValue(detailNode), 0, 70, USE_ALL_WIDTH);
add(desc);
} else if(detailNode.getNodeName().equalsIgnoreCase("dc:date")) {
date = new LabelField(getNodeValue(detailNode), 11, 5, USE_ALL_WIDTH) {
public void paint(Graphics g) {
g.setColor(Color.ORANGE);
super.paint(g);
}
};
add(date);
add(new SeparatorField());
} else if(detailNode.getNodeName().equalsIgnoreCase("pubDate")) {
date = new LabelField(getNodeValue(detailNode), 0, 22, USE_ALL_WIDTH) {
public void paint(Graphics g) {
g.setColor(Color.ORANGE);
super.paint(g);
}
};
add(date);
add(new SeparatorField());
} else {
System.out.println("not the node");
}
} else {
System.out.println("not text node");
}
j++;
}
}
i++;
BinNews bin = new BinNews();
bin.setProv(from.getText());
bin.setTitle(ttl.getText());
bin.setDesc(desc.getText());
bin.setDate(date.getText());
binN.addElement(bin);
}
setBinN(binN);
}
//setBinN(binN);
st.close();
d.close();
} catch (Exception e) {
add (new LabelField(e.toString(),LabelField.HCENTER|LabelField.USE_ALL_WIDTH));
System.out.println(e.toString());
}
}
public String getNodeValue(Node node) {
NodeList nodeList = node.getChildNodes();
Node childNode = nodeList.item(0);
return childNode.getNodeValue();
}
}
I try to store all data from an object called BinNews, to a vector called binN. But when I do debugging, I found that BinN has null value, because "binN.addElement(bin)" doesn't work.
Please advise.
First, you don't actually call setBinN until after the while(i < 10) loop completes. So when you say binN.addElement(bin) then binN will be null.
However your setBinN(binN) call doesn't make sense because you're passing in binN and then setting it to itself which isn't going to do anything.
What you can do is have binN = new Vector(); at the top of the constructor and then it won't be null later on. I don't think the setBinN call will be necessary later on if you're adding the BinNews objects straight to binN.